should be around 0.5 (between -1 and 1 according to Gunnar, but that seems just wrong)
generalisation of Brier score to continuous variables. Smaller is better.
Advantage: logarithmic Score penalises underestimating uncertainty heavily. I feel this is what we want.
Drawback: In contrast to the CRPS, the computation of LogS requires a predictive density. An estimatorcan be obtained with classical nonparametric kernel density estimation (KDE, e.g. Silverman1986). However, this estimator is valid only under stringent theoretical assumptions and canbe fragile in practice: If the outcome falls into the tails of the simulated forecast distribution,the estimated score may be highly sensitive to the choice of the bandwidth tuning parameter.In an MCMC context, a mixture-of-parameters estimator that utilizes a simulated sampleof parameter draws rather than draws from the posterior predictive distribution is a better
–> especially problematic I think if we work with traces and only small sample sizes?
Question do we now the posterior distribution of our draws?
1 day ahead prediction
3 day ahead prediction
7 day ahead prediction
14 day ahead prediction
21 day ahead prediction
median Bias across time
Bias across different time horizons
Bias across different time horizons
median CRPS across time
CRPS across different time horizons
median LogS across time
LogS across different time horizons
median DSS across time
DSS across different time horizons
Across the bench, Sparse AR seems the most reasonable (little bias, ok DSS and LogS)
AR1 seems to be very unconfident (and therefore performs well on LogS) AR1 seems to be downward biased.
Bias across models 1 day ahead
CRPS across models 1 day ahead
LogS across models 1 day ahead
DSS across models 1 day ahead
Sharpness across models 1 day ahead
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|---|---|---|---|---|---|---|---|---|
| 1 | bias | AR 1 | 0.0000000 | 0.1800000 | 0.4000000 | 0.4208571 | 0.6400000 | 0.9600000 | 0.2805722 |
| 1 | bias | Semi-local linear trend | 0.0045000 | 0.2600000 | 0.5000000 | 0.4886122 | 0.7000000 | 1.0000000 | 0.2828632 |
| 1 | bias | Sparse AR | 0.0000000 | 0.2600000 | 0.4400000 | 0.4429796 | 0.6000000 | 0.9400000 | 0.2539925 |
| 1 | bias | Student local linear trend | 0.0000000 | 0.2800000 | 0.4600000 | 0.4940408 | 0.7400000 | 1.0000000 | 0.2887849 |
| 1 | crps | AR 1 | 0.0097440 | 0.0293893 | 0.0526451 | 0.0794650 | 0.0890898 | 0.3660674 | 0.0898345 |
| 1 | crps | Semi-local linear trend | 0.0084990 | 0.0180820 | 0.0310165 | 0.0582366 | 0.0626547 | 0.2951930 | 0.0804311 |
| 1 | crps | Sparse AR | 0.0101793 | 0.0239739 | 0.0382058 | 0.0670498 | 0.0746289 | 0.3242686 | 0.0869147 |
| 1 | crps | Student local linear trend | 0.0078844 | 0.0175479 | 0.0276261 | 0.0574862 | 0.0627260 | 0.3423243 | 0.0799965 |
| 1 | dss | AR 1 | -6.7070804 | -4.2726994 | -3.2469240 | -2.7868620 | -2.0327237 | 4.1355028 | 3.5341067 |
| 1 | dss | Semi-local linear trend | -6.8608629 | -5.4140323 | -4.4642409 | -3.1949170 | -3.0414601 | 6.9085930 | 7.2741175 |
| 1 | dss | Sparse AR | -6.2707872 | -4.7646250 | -4.0417495 | -3.2725326 | -2.6254400 | 3.2154416 | 4.1854044 |
| 1 | dss | Student local linear trend | -6.9603421 | -5.6213290 | -4.5726281 | -3.0854304 | -3.0623496 | 8.5178083 | 8.6348196 |
| 1 | logs | AR 1 | -2.3759070 | -1.2771117 | -0.8254659 | -0.3512798 | -0.2124687 | 3.7791279 | 3.3550183 |
| 1 | logs | Semi-local linear trend | -2.4113148 | -1.7454975 | -1.2850647 | 0.1676840 | -0.5942576 | 5.1365309 | 14.1584179 |
| 1 | logs | Sparse AR | -2.2151640 | -1.4355747 | -1.0007693 | -0.4215960 | -0.4075420 | 2.3876052 | 4.8899705 |
| 1 | logs | Student local linear trend | -2.5628558 | -1.8588923 | -1.3665089 | 0.6381316 | -0.5829726 | 7.2153839 | 21.4649779 |
| 1 | sharpness | AR 1 | 0.0221096 | 0.0709010 | 0.1052553 | 0.1373040 | 0.1700109 | 0.4080446 | 0.1137602 |
| 1 | sharpness | Semi-local linear trend | 0.0220793 | 0.0425032 | 0.0709581 | 0.0970184 | 0.1132235 | 0.3527295 | 0.0872100 |
| 1 | sharpness | Sparse AR | 0.0273751 | 0.0597690 | 0.1020642 | 0.1307556 | 0.1481714 | 0.4327618 | 0.1075388 |
| 1 | sharpness | Student local linear trend | 0.0201258 | 0.0389854 | 0.0597141 | 0.0908121 | 0.1074593 | 0.3644611 | 0.0861209 |
AR1 seems very bad in terms of bias and everything.
Sparse AR is the best in terms of crps, AR1 the worst. Sparse AR also best with dss
–> take Sparse AR
All models have a tendency to be downwards biased, the local and semilocal ones tend to do a bit better.
Bias across models 3 day ahead
CRPS across models 3 day ahead
LogS across models 3 day ahead
DSS across models 3 day ahead
Sharpness across models 3 day ahead
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|---|---|---|---|---|---|---|---|---|
| 3 | bias | AR 1 | 0.0000000 | 0.0800000 | 0.3200000 | 0.3870222 | 0.6400000 | 1.0000000 | 0.3241965 |
| 3 | bias | Semi-local linear trend | 0.0000000 | 0.1800000 | 0.4600000 | 0.4798667 | 0.7750000 | 1.0000000 | 0.3306509 |
| 3 | bias | Sparse AR | 0.0000000 | 0.1600000 | 0.4000000 | 0.4207111 | 0.6400000 | 0.9800000 | 0.3044165 |
| 3 | bias | Student local linear trend | 0.0000000 | 0.2200000 | 0.4500000 | 0.4877333 | 0.8000000 | 1.0000000 | 0.3268705 |
| 3 | crps | AR 1 | 0.0171687 | 0.0614331 | 0.1103108 | 0.1825217 | 0.2175170 | 0.8051064 | 0.2065734 |
| 3 | crps | Semi-local linear trend | 0.0201682 | 0.0477773 | 0.0924898 | 0.1629336 | 0.1807126 | 0.8202050 | 0.2113492 |
| 3 | crps | Sparse AR | 0.0206726 | 0.0506307 | 0.0928446 | 0.1560582 | 0.1823767 | 0.7172435 | 0.1885933 |
| 3 | crps | Student local linear trend | 0.0163728 | 0.0477265 | 0.0863423 | 0.1690528 | 0.1992893 | 0.9294194 | 0.2305773 |
| 3 | dss | AR 1 | -5.9276533 | -2.9450615 | -1.7709012 | 0.6555864 | -0.2748510 | 25.7626906 | 12.1364841 |
| 3 | dss | Semi-local linear trend | -5.1714026 | -3.4990369 | -2.2604842 | 1.0410709 | -0.5360296 | 22.6703036 | 19.6737737 |
| 3 | dss | Sparse AR | -5.1341712 | -3.3923785 | -2.2980771 | -0.6108254 | -0.7036617 | 16.5860866 | 7.2176870 |
| 3 | dss | Student local linear trend | -5.4740366 | -3.4486021 | -2.4468675 | 7.1027040 | -0.5750851 | 33.8384259 | 128.8999208 |
| 3 | logs | AR 1 | -2.0104986 | -0.6239897 | -0.0319122 | Inf | 0.7560601 | 26.4391328 | Inf |
| 3 | logs | Semi-local linear trend | -1.6811899 | -0.7726919 | -0.1932857 | Inf | 0.7073339 | 21.2044455 | Inf |
| 3 | logs | Sparse AR | -1.7106809 | -0.7202203 | -0.2495037 | 1.3074993 | 0.5516765 | 16.3867174 | 9.0427728 |
| 3 | logs | Student local linear trend | -1.8849301 | -0.7960376 | -0.2705402 | Inf | 0.7166684 | 71.4039034 | Inf |
| 3 | sharpness | AR 1 | 0.0259101 | 0.1093206 | 0.1716067 | 0.2187139 | 0.2829760 | 0.6810136 | 0.1803758 |
| 3 | sharpness | Semi-local linear trend | 0.0358389 | 0.0953194 | 0.1487791 | 0.1922980 | 0.2379244 | 0.6989025 | 0.1579592 |
| 3 | sharpness | Sparse AR | 0.0448981 | 0.1150896 | 0.1658032 | 0.2166656 | 0.2701764 | 0.6841201 | 0.1613564 |
| 3 | sharpness | Student local linear trend | 0.0255382 | 0.0886775 | 0.1449848 | 0.1864862 | 0.2316276 | 0.6289547 | 0.1665941 |
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|---|---|---|---|---|---|---|---|---|
| 7 | bias | AR 1 | 0.0000000 | 0.0250000 | 0.2800000 | 0.3428108 | 0.5800000 | 1.0000000 | 0.3221013 |
| 7 | bias | Semi-local linear trend | 0.0000000 | 0.1200000 | 0.4100000 | 0.4577838 | 0.8150000 | 1.0000000 | 0.3509869 |
| 7 | bias | Sparse AR | 0.0000000 | 0.0650000 | 0.3600000 | 0.3907568 | 0.6400000 | 1.0000000 | 0.3230276 |
| 7 | bias | Student local linear trend | 0.0000000 | 0.1400000 | 0.4400000 | 0.4797297 | 0.8200000 | 1.0000000 | 0.3453366 |
| 7 | crps | AR 1 | 0.0224146 | 0.0939478 | 0.1810374 | 0.3033660 | 0.3944736 | 1.1650918 | 0.3233669 |
| 7 | crps | Semi-local linear trend | 0.0408044 | 0.1136825 | 0.2071928 | 0.3195261 | 0.4456299 | 1.1783405 | 0.3019611 |
| 7 | crps | Sparse AR | 0.0411224 | 0.0869667 | 0.1639003 | 0.2659190 | 0.3420645 | 0.9273282 | 0.2865941 |
| 7 | crps | Student local linear trend | 0.0366862 | 0.1256516 | 0.2213939 | 0.3495900 | 0.4473379 | 1.3712852 | 0.3557144 |
| 7 | dss | AR 1 | -5.1511479 | -1.9594018 | -0.9830888 | 9.6924487 | 2.0741012 | 140.8036503 | 50.5344641 |
| 7 | dss | Semi-local linear trend | -3.6826454 | -1.9456928 | -0.8493304 | 4.4328664 | 1.3250952 | 34.6275300 | 32.3297634 |
| 7 | dss | Sparse AR | -3.7060228 | -2.4520010 | -1.2548468 | 2.4033523 | 1.0483922 | 34.4984516 | 12.6638976 |
| 7 | dss | Student local linear trend | -3.6042707 | -1.6218931 | -0.7108610 | 6.2082494 | 1.2599417 | 48.4292621 | 48.2765050 |
| 7 | logs | AR 1 | -1.6846635 | -0.1356936 | 0.4478093 | Inf | 2.0985518 | 287.2871496 | Inf |
| 7 | logs | Semi-local linear trend | -0.9164213 | 0.0317563 | 0.5867681 | Inf | 1.5800334 | 188.5458002 | Inf |
| 7 | logs | Sparse AR | -0.8712581 | -0.2524683 | 0.3474695 | 4.8517056 | 1.3516859 | 43.3777326 | 24.7326588 |
| 7 | logs | Student local linear trend | -1.0542115 | 0.1465011 | 0.6724498 | Inf | 1.6191972 | 586.2568172 | Inf |
| 7 | sharpness | AR 1 | 0.0260282 | 0.1413464 | 0.2472176 | 0.3073751 | 0.3930559 | 0.9850034 | 0.2636063 |
| 7 | sharpness | Semi-local linear trend | 0.0000000 | 0.1636543 | 0.2814537 | 0.3272691 | 0.4235071 | 0.8840668 | 0.2361977 |
| 7 | sharpness | Sparse AR | 0.0601856 | 0.1622306 | 0.2385763 | 0.2984173 | 0.3420976 | 0.8644332 | 0.2239063 |
| 7 | sharpness | Student local linear trend | 0.0000000 | 0.1693309 | 0.3359365 | 0.3761931 | 0.5000669 | 1.0860822 | 0.2989137 |
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
Bias across model forecasts
| horizon | score | model | bottom | lower | median | mean | upper | top | sd |
|---|